The developments described in this section are known to the inventors. However, unless otherwise indicated, it should not be assumed that any of the developments described in this section qualify as prior art merely by virtue of their inclusion in this section, or that those developments are known to a person of ordinary skill in the art.
Distributed computer software products involve multiple applications running on multiple computer processing systems. For example, a simple client-server product involves a client application to be run on a client machine and a web/application server running on a server machine.
More complex software products can require even more applications. As one example, consider the Jira software product commercially available from Atlassian. Running Jira requires a Jira server (or multiple Jira servers if running in a clustered configuration), a database, a Lightweight Directory Access Protocol (LDAP) engine, and a client-side application (e.g. a web browser). To provide flexibility, Jira supports/can be configured to work with a number of different databases (including, for example, Oracle, MySQL, PostgreSQL, Microsoft SQL Server), a number of different LDAP engines (including, for example, Microsoft Active Directory, Apache Directory Server, Apple Open Directory, FedoraDS), and a number of different client-side web browsers (including, for example, Chrome, Safari, Internet Explorer, Firefox, Safari). Overlaying this, several versions of Jira may be in use/supported at any given time. For example, at the time of filing this application, versions 7.7, 7.3, 7.4, 7.5, 7.6, 7.7 and 7.8 of Jira are supported.
As the complexity of a software product increases, so too does the complexity of properly testing new developments before releasing them.
Returning to the Jira example, consider a developer who has developed a new feature (either to the core Jira software product or an add-on/plug-in/integration that is intended to work with Jira) or bug-fix. Depending on the nature of the feature/fix, thorough testing could potentially involve testing the feature for all combinations of Jira version/database/LDAP engine/web browser. For example, a feature that is stable for Jira version 7.7 running on MySQL will not necessarily mean the same feature is stable for Jira 7.7 running Oracle.
In order to test software products such as Jira, test environments are typically set up using virtualization or containerization tools. Properly setting up even a single test environment can, however, be a difficult and time consuming task—let alone setting up multiple test environments to account (for example) for software product variations.